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ABSTRACT 



This external evaluation report provides formative 
evaluation data on several aspects of a project at the University of Northern 
Colorado (UNC) to implement performance-based evaluation of student learning. 
The first section evaluates feedback from 12 senior mathematics and 4 senior 
chemistry majors regarding their understanding of the mathematics standards. 
The evaluation found that the mathematics majors had little knowledge of 
standards -based education and that students wanted benchmarks that would 
indicate whether they were meeting department performance standards 
throughout their undergraduate years. The second section reports students' 
evaluations of the speech and communication department's capstone course and 
departmental expectations of students. While the student comments are varied, 
they stress the importance of freshmen being informed of the requirements of 
the final capstone course. The third section contains informal reports from 
the evaluator to the project director. These focus on interviews with speech, 
chemistry, and Spanish professors involved with the project and working to 
establish standards, capstone courses, and standards-based assessment 
procedures. The final section notes other project activities, including 
dissemination, institutionalization of the project, and writing UNC 
commencement standards and rubrics. Appended are a meeting agenda and 
standards on cultural understanding for Spanish majors. (DB) 



******************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 



o 

ERIC 



"PERFORMANCE-BASED EVALUATION OF STUDENT LEARNING: 
A CONTINUUM FROM K-12 THROUGH THE UNIVERSITY LEVEL" 



Tt" 

Tf 

00 

a 



FIPSE PROJECT #P116A20142 
UNIVERSITY OF NORTHERN COLORADO 

Dr. Francis A. Griffith, Project Director 

Evaluation Report: 4/93 to 8/94 



(V\ 







Prepared by 
A. Rose Shaw 
External Evaluator 




loccs irs 

□ Thie H CENTER ^ ER,C ) 

ssi aa aa 

originating it. ° r gamzatio 

□ Minor changes have been made to 
improve reproduction quality. 



* ^terrsirr in ,his 

official OER, poJS or n> 



y 



BEST COPY AVAILABLE 



A. Rose Shaw, Ph.D 



METRICA 
P.0. Box 5167 
Greeley, CO 80631 




3 



A PROCESS EVALUATION REPORT OF THE FIPSE PROJECT 



"PERFORMANCE-BASED EVALUATION OF STUDENT LEARNING: 
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TOPIC: EVALUATION OF STANDARDS BY 



SENIOR MATHEMATICS MAJORS AT UNC 
SENIOR CHEMISTRY MAJORS AT UNC 



INTRODUCTION 



During spring semester 1994, twelve senior mathematics 
majors and four senior chemistry majors were asked to evaluate 
the Mathematics Standards. One project goal is to improve the 
preparation of UNC teacher-education graduates for entry-level 
positions in outcomes-based school systems, such as School 
District #6. One program objective is to include students, 
faculty, and community members in the formulation and evaluation 
of UNC Commencement Standards. The evaluation of the standards 
and the feedback from mathematics and chemistry majors at UNC is 
one of the steps in the process of meeting these goals and 
obj ectives . 

FEEDBACK FROM MATHEMATICS STUDENTS 
Responses are summarized below: 

1) Do you know what graduation standards have been established in 
your department? 

10 responded "no" 

2 responded "yes" 

t 

2) If yes, what are they? 

Responses were: "Cum 2.50 gpa, 48 semester hours of math 

content" and "2.50 GPA. Selected courses consisting of 48 
semester hours." 
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At this time the students were given copies of the 
Mathematics Standards developed by UNC and School District #6 
faculty. They were asked: 

3) Are there any parts that you don't understand, don't agree 
with, or which need further clarification? 

One student responded, "This seems a bit ambiguous and quite 
subjective. But, in general, there needs to be concrete 
standards . " 

4) Do you think students should be able to do these things by 
the time they graduate? 

r 

All 12 students responded, "Yes." 

5) Would you have found these helpful when you decided to choose 
your academic major? 

Nine students responded, "Yes." 

Three students responded, "No." 

6) Suppose a senior is told, "You cannot graduate because you 
haven't met these standards." What is your reaction to this? 

Comments were (direct quotes) : 

"If the standards changed mid-stream I'd be pXXXXX." 

"The senior should not have been allowed to get that far 
until the "bombshell" is dropped. Students should be required to 
meet requirements at certain periods and this should be 
monitored. " 
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"WRONG From my experience half the mistakes or problems are 
caused by UNC and faculty. " 

"Students need to make sure they settle things before they 
are seniors . " 

"I hope it's not me. Make sure students fully understand as 
freshman: the standards." 

"As a senior, teachers should be teaching students so they 
are prepared to pass the standards. It's not fully the student's 
fault . " 

"As a senior, I would be angry. These standards should be 
made known at an earlier time." 

"It depends on the circumstances." 

"Why wasn't I told this before?" . 

"Why wasn't anything mentioned at my advising appointments 
until my last semester?" 

"Standards should be phased in so freshmen can be given the 
requirements they need to fulfill." 

"Why didn't someone tell me this 4 years ago?" 

If you could show these standards to a potential employer, do 
think it would enhance your opportunities for employment? 

7 responded, "Yes." 

4 responded, "No. " 

1 no response 

Please note any further feedback you have for us... 

Two students commented. One said, "The standards should be 

4 



7 ) 

you 



8 ) 



O 

ERIC 



8 



an integral part of every class. Getting some entrenched 



professors to do this will be close to impossible . " The other 
student said, "There are far too many things to do to graduate as 
a teacher. No other major requires so much for so little pay." 



FEEDBACK FROM CHEMISTRY 'MAJORS 

During spring semester 1994 four chemistry majors (there are 
very few chemistry majors at UNC) completed the same 
questionnaire the twelve mathematics majors had completed. 
Responses included: 

1) 1 responded, "Yes" and 3 responded, "No." 

2) The response to this item by the one student who had said 

"Yes," was: "Pass the required classes for your major and all 

the general education classes." 

3) 1 responded, "No." There were three none responses. 

4 ) All four responded, "Yes." 

5) Two replied, "Yes," and two replied, "No." 

6) "That senior should not graduate until the standards are 
met . " 

"It is appropriate to gauge a student's progress by 
standardized testing, but what is learned cannot be fully 
disclosed from the results of these tests. There should be 
opportunities for the student to improve in the areas of 
deficiency without denying the student graduation. Grades should 
mean something, too." 
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"Providing the senior has been in the standards -based 
education I think they shouldn't graduate." 

"If a person is graduated that is not proficient in his/her 
field it brings down the department, school, and field. If the 
person cannot perform these tasks he should not graduate at all . " 

7) All four responded, "Yes." 

8) There were no additional comments by these students. 



DISCUSSION 

It appears from the written feedback from the chemistry 
students that they have a better understanding of standards-based 
education than the mathematics students do. The feedback from 
the mathematics majors suggests little knowledge of standards- 
based education. Being exposed to this questionnaire will 
heighten the UNC math majors' awareness of SBE which will be 
beneficial if they decide they want to teach in a district that 
turns out to be an SBE district. 

In general, the students felt that if there are department 
standards, then benchmarks should exist that tell students 
throughout their undergraduate years at UNC whether or not they 
are meeting department performance expectations. Although the 
students did not know the terminology, "benchmark" , this is what 
many of them referred to in their comments. 

One math major's comment about the difficulties that might 
exist among "entrenched faculty" in the conversion to SBE is 
interesting. Perhaps it is a prophetic remark. It is important 
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for project staff to be aware of resistance that might exist in 
the School District and among UNC faculty members. 
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TOPIC: STUDENT EVALUATION OF 

SPEECH AND COMMUNICATION'S CAPSTONE 
COURSE AND DEPARTMENTAL EXPECTATIONS 
OF STUDENTS 
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INTRODUCTION 



Appropriate curricular changes, the need for standards, the 
function of capstone courses, and feedback from students are all 
intertwined in the objectives of the project. One Speech and 
Communication's teacher asked his capstone course students for 
feedback on May 12, 1994. 

DISCUSSION 

The summary of the evaluation that was FAXed to the Project 
Director by the teacher has been copied and is included in the 
following pages . 
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The following data was gathered on Friday, May 12, 1994 during the 
final "Capstone" course session in the UNC department of Speech 
Communication. 

The approximately thirty (30) seniors graduating from the 
department were broken into four focus groups consisting of 
approximately seven (7) students each. Each graduating student in 
the department was familiar with the commencement and 
performance standards proposed by the department. 

Joe Downing, a representative of Project S.P.A.N, Dr. Warnemunde, Dr. 
Arneson and Dr. Allen (all faculty members of the SPCO department) 
rotated facilitation duties of each group. Joe Downing, then, 
interacted with each focus group for approximately twenty minutes. 

Joe Downing introduced himself as an alumni of the department 
(M.A., Human Communication) as well as a representative of Project 
S.P.A.N. The goals of the project were delineated. Each focus group 
was asked to comment on how they felt about the education they 
received in the UNC SPCO department. 

What follows are the notes Joe Downing made on the student's 
comments. They are written in the order they were received: 

• the department offered a good theoretical base 

• not enough practical analysis of the theory was given 

• the internship was not stressed 

• classes are too big 

• interpersonal communication strategies are stressed, which is 
good 

• students need to be told when they enter as first-year students 
that they will be tested bn the information as seniors 

• Professors need to tell students as freshmen that they will need 
to keep their notes for a Capstone course 

• the curriculum lacks on the practical side. More role-playing is 
needed, especially in the organizational communication course 

• not enough public speaking "demanded" 

• the goals and objectives of the department need to be given to 
students as freshmen 

• there is no continuity between the professor on departmental 
goals 

10 
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• nol enough emphasis is placed on how to write (especially the 
structure) 

• students need to see the big picture as freshmen 

• education emphasis is not emphasized in the Methods class 

• communication and influence is the strongest part of the 
department 

• small group and organizational communication courses nol 
emphasized 

• there needs to be some pre-requisites to be admitted in the 
department 

• the communication department is thought to be the easiest on 
campus. There were loo many people in Karre's Interpersonal 

— - course (everyone iB the particular focus group strongly agreed 
with this statement) 

• Courtroom Communication course best in department. Liked 
the team critique approach. Most practical course in 
department 

• need more structure in department, teachers do not work 
together 

• the internship was not stressed 

• need restricted registration in courses 

• diverse classes are good, as are different teaching styles 

• need more emphasis on organizational communication 

• communicating as a manager not taught, not enough applied 
communication. Not taught how to communicate with a boss 

• mediation and arbitration courses not taught 

• conflict mediation skills not taught 

• same teachers take the applied notion too strong; all you do is 
role playing in some courses 

• curriculum not consistent between community colleges and UNC 
SPCO department. They use different assessment measures in 
the two sets of institutions. There are lots of transfer students 
here from community colleges 

• cannot synthesize the course information until the Capstone 
course 

• how does a person who does well in every class fail a test in the 
Capstone course? 

• need more required computer courses 

• leadership course very applicable 

• what is tested in the Capstone course isn't necessarily tested 
in the particular course 

• developing skills nol emphasized, we learned the titles of 
concepts and their theory, but we couldn’t apply them 
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• conflict class was good - very practical 

• more skills need to be developed 

• applying concepts was done well in the department 

• more of an emphasis needs to be placed on inter cultural 
communication 

• these are invaluable life skills we are learning 

• more skills need to be emphasized 

• speech courses are too easy; a minimal effort is all that is 
needed to earn a "B" 

• speech classes thought around campus as "G.P.A. boosters" 

• the interaction in the classes is good, students are allowed to be 
as extroverted as they feel comfortable 

• conflict management needs a lot more emphasis 

• what do you do with this degree? There is no direction given 
by the faculty 

• why don't we have a career board like the Journalism 
department? 

• speech communication internships don't exist. There is too 
much red tape. You have to go out on your own to find an 
internship 

• Career services have no idea what to do with a speech major 

• speech department has a good reputation within the UNC 
community. Business majors think the speech major is too 
easy 

• more general education courses needed 

• not enough public speaking is emphasized in the department 

• it was good to learn the different facets of communication 

• feels like 1 have the skills to prepare a speech 

• need to apply the knowledge; you can analyze conflict but you 
can’t "fix it" 

• department has given her the confidence to speak in public. 

Most freshmen are shy when they enter the department 

• more emphasis needs to be placed on written communication 

• not enough writing skills are emphasized. We all took the bare 
minimum of English courses 

• I am graduating with too general a degree. I need more 
application with specific skills 

• need to apply the class content to future student goals 

• need more public speaking 

• can't apply the public speaking outside of the department (she 
used the example of Toastmasters) 

• we learned theories three years ago, but we don’t remember 
them now 

It 
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DISCUSSION 



Evaluation of the project included External Evaluator 
interviews with Speech, Chemistry, and Spanish professors 
involved with the project. These are three of the five UNC 
departments working to establish standards, capstone courses, and 
standards -based assessment procedures. The quality of the 
involvement of these departments is integral to the project's 
realization of its goals and objectives pertaining to the 
establishment of SBE on the UNC campus. Informal reports sent to 
the Project Director by the Evaluator follow. 
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June 9, 1994 

TO: DR. JERRY GRIFFITH 

FROM : ROSE SHAW 

RE: DISCUSSION ROSE HAD WITH DR. RICH SCHWENZ REGARDING PROJECT 

SPAN ON JUNE 9, 1994. 



The ETS Exam has been taken by the department's seniors since 1988. 
Prior to 1994, there was little motivation for the seniors taking 
the test to excel. The test has not been a requirement. Students 
volunteer to take the ETS Exam. This year motivation was higher 
for the seniors because the department announced that each student 
would have a folder which could include the ETS scores. Some of 
the students studied for the test which, as far as Rich knows, has 
never happened before spring 1994. 

In Physical Chemistry, the seniors were given a rubric for grading 
one of their experiments. The students said they liked the 
"contract grading" scheme and modified their plans to conform to 
the rubric. Unfortunately severe senioritis set in the final weeks 
of the semester and the reports they wrote were not as good as they 
probably could have been. The PC students do give oral reports; no 
one reported on the rubric -driven lab. 

I asked Rich what he thought the department might be concentrating 
on in the fall as far as standards -based education is concerned. 
He indicated that he hasn't thought this through but that the 
department needs to resolve the dichotomy between normative-based 
testing (ETS and ACS) and standards -based education so he expects 
this is what they will be concentrating on during the fall 1994. 

On February 25th, Rich gave a department seminar about standards- 
based education and SPAN. All the department faculty and graduate 
students attended. There were also two faculty members present 
from Earth Sciences. The department faculty members are beginning 
to realize that HB1313 is going to have an effect on them. 

I asked Rich for his suggestions for informing faculty members 
about SPAN. He indicated that he thought department chairs should 
be targeted on a one-to-one basis especially those heavily involved 
in teacher education. He thinks the contacts with faculty members 
and department chairs should be done by faculty members rather than 
by someone from administration. Rich volunteered to talk to the 
chairs in Ross Hall. I told him you would be in contact with him 
about doing just that. 



June 15, 1994 



TO: DR. JERRY GRIFFITH 

FROM: ROSE SHAW 

RE: REPORT ON DISCUSSION WITH DR. LYNN SANDSTEDT REGARDING PROJECT 

SPAN on June 15, 1994. 



Dr. Sandstedt indicated that he never was able to get the members 
of his department to cooperate on the plans they had for testing, 
etc. as SPAN participants. He hopes to have better participation 
in the fall. 

I wonder if it would help Dr. Sandstedt' s faculty to become more 
committed if a faculty member (how about Rich Schwenz) could speak 
at one of the early fall Hispanic Studies Department meetings. 
Perhaps if he told the members of that , department how important 
SPAN is and how it connects to HB1313, etc. that Dr. Sandstedt 's 
faculty would be a little more cooperative. Perhaps you might 
suggest this to Lynn. I did not suggest anything. As the 
evaluator I simply listened and am reporting our conversation to 
you . 



July 6, 1994 



TO: Dr. Jerry Griffith 

Project SPAN 

FROM: A. Rose Shaw 

Project Evaluator 

RE: Meeting with Dr. Pat Arneson regarding Capstone course - 

UNC's Department of Speech Communication 



Today I met with Dr. Pat Arneson in her office in Candelaria 
Hall on the UNC campus. We talked in length about the Capstone 
course in the Speech Communication Department . As you know, she 
has been the professor in the department teaching that course 
since 1990. I was impressed with the commitment Pat and the 
Department have made to designing, implementing, and constantly 
improving the Capstone course. 

Their majors know where they stand right from the beginning. 
The Department has carefully documented what students should 
learn in each Speech Communication course and professors teaching 
the courses are suppose to teach to the document. This, of 
course, runs into trouble spots especially since the department 
relies on temporary instructors to teach many of the freshman- 
level courses. The department has learned that there are many 
details that need to be taken care of such as informing their 
majors when they are freshmen that they need to keep the books 
they used in their Speech Communication classes because they'll 
need them again when they enroll in the Capstone course as 
seniors . This is an example of the kind of detail that is unique 
to departments and areas of emphasis and are often revealed as 
programs and courses are implemented. 

The seniors in the Capstone course must demonstrate 
competency in three areas: (1) Interpersonal communication (2) 

Small group communication, and (3) Communication and influence. 
The seniors are tested in each of these areas both quantitatively 
(written tests) and qualitatively (behavioral performance) . The 
department publishes a student handbook so the seniors know 
exactly what is expected of them and how they will be evaluated. 
The handbook includes a list of 17 possible techniques for 
assessment including argumentation/debate , case study, journals, 
presentations, video tape assignments, and games. There are 
three standards: advanced standard, essential standard, and in- 

progress standard. If a student receives three items of "1", 
then that student participates in an "Extended Learning 
Opportunity . " 

Students need a C or better in the Capstone course to 
graduate. At this time no student has been kept from graduating 
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but Pat said the course is known as a rigorous course and the 
students work very hard in it. The three compentency areas 
account for 30% of the class grade and material. The department 
has designed this course to include integration of new 
information as well as recall of old information. 

During the week of final exams, the seniors in this course 
participate in an exit interview where they offer their 
perceptions of the department including the strengths and 
weaknesses of their program of study. 

I asked Pat if their plans for this fall include any changes 
in the Capstone course. She indicated that her plans for the 
course include 

(a) Providing more specific, refined rubrics, 

(b) Having one professor other than herself grade each set 
of 

tests in each of the areas of competency, and 

(c) Breaking the course into three 4 -week segments so that 
she can devote one month to each of the three competency 
areas . 

The enrollment in the Capstone course is usually about 30 to 
35 although they had only 17 students in the course last fall. 

Pat found the smaller class size much easier to teach and the 
students said they liked the small class size. But a class of 
only 17 accounted for low F.T.E. in the department so small 
classes are not possible because of the need to produce F.T.E. 

Pat did acknowledge, after I asked her, that teaching the _ 
Capstone is very time consuming, in part, because of the time it 
takes to, do authentic assessment. She estimated that teaching 
the Capstone course is equivalent to teaching two regular 
classes. It has helped her to have other professors grade the 
written exams but teaching the course still takes a great deal of 
time. For example, Pat estimates that it takes two weeks for her 
to do authentic assessment in one area. 

It is my. impression that teaching the Capstone course is not 
easy because it's a hard course for students, they probably feel 
threatened, scared, and more than a little "put upon" . The 
course appears to be a big challenge for both the teacher and the 
students. But at the end, when the seniors have passed the 
course, they can say with pride,' "I know I am a competent 
communicator." That is probably one of the most rewarding 
outcomes of the Speech Communication Department's Capstone 
course. 
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ERIC 

hfliflaffBEaaaa 



23 



INTRODUCTION AND DISCUSSION 



Much of the project activity was devoted to planning 
strategies for institutionalizing the project. These have been 
timer consuming activities that have not necessarily shown 
immediate results but are instrumental to the success of this 
project . 

PROJECT ACTIVITIES 

■3 

Objective: Dissemination. Three graduate students assigned 

to the project began working on the development of a SPAN 
brochure. Spring 1994. . 

Objective: Institutionalization of the project. The Project 

Director, Dr. Griffith, met with several University faculty and 
administrators regarding their suggestions about ways to include 
more faculty in project activities and to familiarize faculty 
with SBE and performance-based assessment.. Dr. Griffith's spring 
1994 meeting with Dr. King from the Educational Leadership 
Department resulted in a plan for linking the project with the 
newly University' s formed Tointon Institute for Educational 
Change. Dr. Griffith met with the Chair of Speech 
Communication, the Dean of the College of Arts and Sciences, 
and the Assistant Vice Provost for Academic Affairs during the 
summer 1994. 

Objective: Writing UNC Commencement Standards and rubrics. 

The Project Director met with two Leadership Faculty ( Spanish 
Department) to discuss evaluating standards and rubrics. The 
agenda for this meeting and the Standards are in Appendix I. 
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Agenda for Meeting with Lynn Sanstedt 
and Terry Ballman, Friday, July 29, 1994 

The standards for the study of culture seem to be: 

a. Geography and Economics -v ( 

b. History of Spain and Latin America /"Oj-t-ts - 

c. Artistic and literary contributions (of Spa in and Latin Anre rrca?) 

d. Daily customs and routines of social, professional, and technical 
discourse. 

Are these accurate? 

We need a descriptor for each standard across each level. For example, 
there are five descriptors under Novice, four under Intermediate and three 
under Advanced and Superior. 

What task(s) will the scoring rubric be applied to? Can all standaids be 
assessed within the context of a single task? If not, how many tasks will 
be needed, what will be their nature and when will they be done? 

What about a standard(s) pertaining to critical thinking or problem- 
solving. What kind of rubric is needed and where and how can these 
dimensions be evaluated? 

What are some means to obtain feedback about the standards and rubrics 
from : 

a. Hispanic Studies faculty 

b. entering majors 

c. exiting majors , . 0 

d. external parties, including employers of program graduates 
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